Running Head: EDUCATING AT-RISK AFRICAN AMERICAN MALES 



Educating At-Risk African American Males: Formative and Summative Evaluation of the Street 

Academy Program 

Judi Vanderhaar, Ph.D. 

University of Louisville 
Marco A. Munoz, Ed.D. 



Jefferson County Public Schools 



Abstract 



No Child Left Behind legislation has triggered increased attention toward closing the 
achievement gap. Of particular importance is the need to improve African American male 
outcomes in education. The Jefferson County Public School System (JCPS) has embarked on one 
such effort with the implementation of the Street Academy Pilot Program. Evaluation of the 
after-school program included formative and summative components. A double pretest-post 
design with an internal control group was used to assess the impact of the program on students’ 
reading achievement, school attendance, grade promotion, school discipline and, self-esteem. 
Significant effects were not seen for every measure, but the consistent direction of the effects 
showed overall academic benefits for the participants. Implications for practice and policy are 
discussed. 

Keywords: Academic Achievement; Accountability; Educationally Disadvantaged; Evaluation; 
Federal Legislation; Minority Group Children; Poverty; Racial Differences. 
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Introduction 



Jefferson County Public Schools (JCPS) is the 13 th largest urban school district in the 
United States (Standards & Poor, 2005). The school district serves more than 96,000 students 
from Pre School to grade 12. JCPS has a vision for long term student achievement for all 
students. This vision commits JCPS to continue closing the long standing achievement gap 
between its African American and White students. Further, JCPS strives to ensure that all 
students meet high academic standards. A good deal of this effort includes helping students excel 
in the area of literacy, reading on grade level, and providing attention to non-cognitive student 
measure such as attendance and discipline. While JCPS has made great strides in closing their 
achievement gap, the wider social realities and trends experienced by young African American 
males, necessitate the strengthen of this critical effort. 

A recent New York Times article explained that emerging studies by experts at 
Columbia, Princeton and Harvard Universities indicated that the rapidly increasing population of 
poorly educated Black men is ‘becoming ever more disconnected from the mainstream society’ 
(Eckholm, March 20, 2006, p.l). For example, one study shows that in 2004, 72% of Black 
male high school dropouts in their 20’s were jobless. Further, 6 in 10 of Black men in their mid- 
30’ s who had dropped out of school had spent time in prison. One of the top recommendations 
that emerge from the studies is a call for ‘intensive new efforts to give them a better start that 
including support for parents and extra schooling’ (p.5). 

While the vast scope and serious condition of African American male outcomes in 
education is devastatingly evident, many efforts to alleviate and improve this condition are 
abound in community and educational sites around the nation. A key, researched based strategy 
includes providing out of school programs for African American males. Riordan (2002) asserts 
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that Single sex educational environments tend to be more disciplined and academically rigorous, 
particularly for disadvantaged students. Previous research that examines after school programs 
for Black male students’ shows that such programs showing effectiveness all include specific 
goals and have an evaluation component (Fashola, 2005). Providing an all African American 
male setting during the nonschool hours is one way to concentrate on developing the talents of 
young African American males (Fashola, 2005) and can accelerate the academic and 
psychosocial improvement of African American males. 

The Street Academy is one such effort that JCPS has invested in. The Street Academy 
model was designed by the New York City Urban League with the goal of providing minority 
male students in poverty a supplemental program using a holistic framework. The Street 
Academy includes both specific goals and its evaluation has a formative and summative 
components. Though in its infancy stage (i.e., phase I pilot), the Street Academy program has 
tremendous potential to impact the educational experiences and outcomes of young African 
American males within the JCPS system. 

Program Description 

The Street Academy concept is a three (3) phase initiative designed to address the 
growing need for working with African-American male students with a multi-faceted/holistic 
and neighborhood-based approach. Phase I was designed to operate on Saturday and is similar to 
the Louisville Urban League’s (LUL) Rising Fifth Grader Program 1 ;Phase II will entail the 
Saturday activities plus working with students for half day or longer during the week in a school 
setting; and, Phase III will be a long term process of serving students at high risk of school 
failure, in an all African American male residential, boarding school type setting situated in the 
inner-city This evaluation centers on the processes and outcomes of Phase I. 
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The street academy operates from a holistic approach that focuses on four major 
components: academic, socio-behavioral, community support, and culture. The academic 
component will involve a combination of the District’s literacy initiative and Direct Instruction. 
Academic component emphasizes reading and utilizes the Corrective Reading instructional 
model; the Corrective Reading is a highly structured Direct Instruction reading intervention 
program that includes both decoding and comprehension strategies: (a) word attack, (b) story 
reading, (c) reading checkouts, (d) workbook, and (e) checking workbook. The curriculum also 
involves a cultural component that highlights African American literature. 

The instructors are African-American male JCPS teachers who have been trained in the 
Corrective Reading model and have a passion for working with low income minority students. 
The program emphasizes the cultivation of appropriate behavior by transmitting to students a 
specific code of conduct coupled with high expectations. Included in the holistic approach is the 
community/support component facilitated by a home school coordinator who works with the 
parents or principal caregiver of the students. 

The Street Academy operates on Saturdays at Wheatley Elementary School. African 
American male students who live in high poverty areas (the Wheatley area) and attend Title I 
school participate in the program. Transportation is provided to students as needed. The program 
operated from December 4 - the end of May 20, 2006 and included 16 Saturday sessions. The 
sessions run from 1:30 - 4:30 with two hours dedicated to intense academic instruction. 



Hhe Louisville Urban League (LUL) is a local affiliate of the National Urban League, a private, non-profit 
social services and civil rights-related organization. The LUL Rising Fifth-Grader Program has been in 
operation for 10 years and focuses on low-performing students from poverty and centers on teaching 
literacy, primarily. It meets on Saturday during the school year and half-a-day during the summer. 
Evaluation data indicate thatthe program hasbeen successful in raising reading sc ores of participants 
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Program Goals & Objectives 

The goals of the Street Academy is to assist African American males in grades 4-6 attain 
proficiency in reading, increase school attendance, and reduce negative behavior in school. 
While the non-cognitive goals of the Street Academy are critical, the primary focus of the Street 
Academy is to increase the literacy levels of the participants. 

By providing a structured, nurturing environment conducive to the culture of the 
participants, individualized student attention, and home/school communication that includes a 
focus on academics, socio/behavioral, community and culture, the Street Academy strives to 
accomplish six specific objectives: (a) increase the percent of students scoring 
proficient/distinguished on the Predictive Assessment Series (PAS) in reading; (b) decrease the 
percent of students scoring novice on the PAS in reading; (c) ensure the successful promotion of 
6 th grade participants into 7 th grade; (d)increase participants reading grade equivalency on the 
Gates-MacGinitie test; (e) increase school attendance as measured by reduction in absences and 
tardies; and, (f) reduce negative behavior as measured by a decrease in school disciplinary 
referrals and suspensions 
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Evaluation Purpose & Questions 



This Street Academy Program Phase I evaluation report includes both formative and 
summative dimensions. The project coordinator will use the information provided in this 
evaluation to assess the formative and summative aspects of the Street Academy Phase I project 
for program improvement purposes. The three fold evaluation objectives include conducting a 
formative evaluation, a summative evaluation, and a cost analysis. 

Formative Evaluation 

The first objective was to conduct a formative evaluation. The formative evaluation 
helps to understand program dynamics and procedures and provides timely quality assurance 
guidance to the project coordinator. The overarching evaluation questions that guided the 
formative study were the following: 

(a) What are the demographic characteristics of the students participating in Street 
Academy? 

(b) What schools do the Street Academy participants attend? 

(c) By what criteria and process are students referred to the Street Academy? 

(d) What is the Street Academy Curriculum and what activities take place? 

(e) Who are the teachers of the Street Academy? 

(f) How often do participating students attend the Street Academy Saturday sessions? 

(g) What progress is assessed through a mid-program feedback assessment? 

Summative Evaluation 

The second objective was to conduct a summative evaluation. A summative evaluation will 
help to assess the impact of the program on participants. The primary goal of the evaluation will 
help assess the degree to which the Street Academy program improves students’ academic 
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performance in reading, discipline, and attendance. Based on the goals and objectives of the 
program, the overarching evaluation questions that guided the summative study were the 
following: 

(a) What is the overall impact of Street Academy on the students’ reading achievement 
as measured using the winter and spring Predictive Assessment Series? 

(b) What is the overall impact of the Street Academy on the students’ reading 
achievement and reading grade equivalency as measured using the Gates- 
MacGinitie test? 

(c) What is the overall impact of the Street Academy on the 6 th grade participants’ grade 
promotion rate? 

(d) When compared to last school year (i.e., December thru May 2004-2005), is there an 
increase in participating students school attendance (i.e., unexcused absences and 
tardies)? 

(e) When compared to last school year (i.e., December thru May 2004) is there an 
improvement in student school discipline as measured by participating students’ 
disciplinary infractions (i.e., referrals, suspensions)? 

(f) What are the participating students’ perceptions of the Street Academy program and 
teachers? 

(g) What is the overall impact of the Street Academy on the participants’ perception of 
themselves in general and perception of themselves as a student? 
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Cost-Analysis 

The third objective of the evaluation was to conduct a Cost Analysis of the Street Academy 
Phase I. The cost analysis includes the program budget and cost per student. This analysis also 
includes the number of Saturday sessions and cost per student per session. With the summative 
results in mind, this analysis assists in determining the cost-effectiveness of the program. The 
questions that guided the cost benefit analysis were the following: 

(a) What are the main components of the Street Academy budget? 

(b) What is the cost per student for all participants served? 

(c) What is the cost per student per session for all participants served? 

(d) What is the cost per student for all participants who attended more than 50% ? 

(e) What is the cost per student per session for all participants who attended more 

than 50%? 
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Evaluation Method 



Participants 

The Street Academy program served a total of 38 African-American male students. As 
shown in Table 2, 42% of participants attend Elementary school and 57% attend Middle school. 
Approximately 42% are in fourth and fifth grade, and approximately 45% are in sixth grade. 
With respect to household structure and poverty the students are at high risk as 71% live in 
single parent homes and 92% receive free/reduced lunch. 

Table 2 

Profile of Participating Students in the Street Academy Program (N = 38) 


Variable 


N 


% 


School Level 






Elementary 


16 


42.1% 


Middle 


22 


57.9% 


Grade 






Third 


1 


2.6% 


Fourth 


9 


23.7% 


Fifth 


7 


18.4% 


Sixth 


17 


44.7% 


Seventh 


3 


7.9% 


Eighth 


1 


2.6% 


Parent Household Structure 






Single 


27 


71.1% 


Dual 


11 


28.9% 


Lunch Status 






Free/Reduced 


35 


92.1% 


Paid 


3 


7.9% 



Note : All Participants are African-American Males 
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The Street Academy program served students who attend school throughout 15 different 



JCPS school locations (seven elementary schools and nine middle schools). All schools attended 
by participants in the 2005-2006 school year met the Title I criterion (i.e., over 50% of student 
population on free/reduced lunch), except one (i.e. 2 participants from Johnson Traditional). Of 
the elementary student participants, approximately 38% attend Wheatley elementary. Of the 
middle school participants, approximately 41% attend Lassiter Middle school. 

Table 3 

School Locations of Street Academy Participants (N =15) 



School Name 


Number 


Percent 


Elementary School Level 


Kennedy Elementary 


1 


2.6% 


King Elementary 


1 


2.6% 


Medora Elementary 


1 


2.6% 


Minors Lane Elementary 


4 


10.5% 


Roosevelt Perry Elementary 


1 


2.6% 


Wheatley Elementary 


6 


15.8% 


Zachary Taylor Elementary 


2 


5.3% 


Elementary Total 


16 


42.1% 


Middle School Level 


Carrithers Middle 


4 


10.5% 


Johnson Traditional Middle 


2 


5.3% 


Kennedy Metro Middle 


1 


2.6% 


Lassiter Middle 


9 


23.7% 


Myers Middle 


2 


5.3% 


Noe Middle 


1 


2.6% 


Stuart Middle 


1 


2.6% 


Western Middle 


1 


2.6% 


Westport Middle 


1 


2.6% 


Middle Total 


22 


57.9% 


TOTAL 


38 


100% 



Note: the two participants from Johnson Traditional were not regular attendees (mean attendance rate = 18.7%) 
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Data Collection and Measures 



The project evaluator used various sources to obtain the qualitative and quantitative data 
used in for the formative and summative evaluation of the Street Academy. The District’s 
computerized database provided individual student demographic, cognitive and non-cognitive 
data. The demographic data included identification number, race, gender, school location, grade, 
free/reduced lunch status, and household structure. The cognitive data drawn from the data base 
included statewide test scores in reading as well as reading scores and proficiency levels on the 
PAS. The PAS is a series of equivalent assessments designed to match, predict, and improve the 
knowledge and skills tested by state standards and aligned to nationally-normed and state 
criterion test. Non-cognitive data provided by the district data base included attendance and 
discipline related information. 

Qualitative process data was collected throughout the formative data collection process 
via site visits and meetings with Street Academy program coordinator and the Street Academy 
educational consultant. This includes information pertaining to referral process, program 
activities, curriculum, and program climate/culture. The specific pre and post test measures are 
shown on table 1. 
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Table 1 



Street Academy Phase I Evaluation Measures 



Cognitive Measure 


Pre-test 1 


Pre-test 2 


Post-test 


Reading 


Predictive Assessment Series (PAS) 


Fall 05’ 


Winter 05’ 


Spring 06’ 


Proficiency Level and Scale Scores 


September 


December 


February 




Pre-test 


Post-test 


Gate s/Mac Ginitie Test 

Raw Scores and Grade Equivalent 


January 2006 


May 2006 


Non-Cognitive Measures 


Pre-test 


Post-test 


Attendance 


Number of School Absences 


Dec - May 2004/2005 


Dec - May 2005/2006 


Number of School Tardies 


Dec - May 2004/2005 


Dec - May 2005/2006 


Discipline 


Number of School Referrals 


Dec - May 2004/2005 


Dec - May 2005/2006 


Number of School Suspensions 


Dec - May 2004/2005 


Dec - May 2005/2006 


Survey 


Perceptions of self in general and self as 
student 


Retrospective (Nov 2005) 


May 2006 


Perception of program and teachers 




May 2006 



Design 

For the purpose of the summative evaluation, a pre - post design with a double pre-test 
and an internal control group was used. Embedded within the overall evaluation design was a 
formative process evaluation to provide mid-program guidance. Employing additional elements 
to a simple pre-post research design, such as a double pre test and an internal or “natural” control 
group enhances confidence in causal knowledge and reduces internal validity threats (Shaddish, 
Cook & Campbell, 2002). 
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With respect to assessing cognitive improvement (the primary aim of the program), a 
double pretest was used. While a pretest helps determine the degree of similarity between the 
treatment and internal control group, having a second pretest helps determine if the groups are 
changing at similar rates, prior to the program. The confidence in causal knowledge with a pre- 
post design is increased when the interval between pre test and post test is short (Shaddish, Cook 
& Campbell, 2002,). 

The internal control group can be considered the “natural” control group. This group was 
drawn from the participants who were referred to the Street Academy program, but for various 
reasons, did not experience the full “treatment.” That is, some participants did not complete the 
program, did not attend the program regularly, or began the program late in the year. The 
“control” students only attended the Street Academy between 6% - 46% of the sessions. The 
“treatment” students attended between 56% - 100% of the sessions. Because these students are 
similar to the “treatment” students and met the same criteria, they will act as a plausible and 
natural control group. 

In addition to providing mid-program feedback based on quantitative data, the formative 
evaluation allowed collection of some quantitative information regarding the program from 
various stakeholders. This design entailed the collection and analysis of participant data 
throughout the duration of the Academy program. The formative evaluation also utilized 
descriptive statistics to describe participants’ demographic characteristics. 

The summative evaluation included both descriptive and inferential statistics. In order to 
analyze academic achievement in reading, descriptive statistics were used to compare the 
number and percent of “control” students who increased in proficiency level versus the number 
and percent of “treatment” students who increased in proficiency level on the Predictive 
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Assessment Series (PAS) between fall to winter and between winter to spring within the 2005- 
2006 school year. Because the intervals between the PAS tests are short, the confidence in causal 
knowledge is increased (Shaddish, Cook & Campbell, 2002). The fall PAS was used as pretest 1, 
the winter was used as pretest 2, and the spring was used as the post test. Additionally, within 
the treatment group, descriptive statistics were used to compare change in level between 
elementary and middle school participants. 

In order to assess an additional measure of participants reading achievement, the Gates- 
MacGinitie reading pre - test was administered at the start of the program (January 2006) and the 
post-test was administered at the end of the program (May 2006). This test measures changes in 
students decoding and comprehension skills. Descriptive statistics of raw scores were analyzed 
and dependent sample t-test was conducted to determine any statistically significant changes in 
scores from pre- test to post-test. Further, descriptive statistics of grade equivalency scores are 
provided to determine any change in grade equivalent reading level of participants. 

With respect to non-cognitive measures, attendance and discipline data were analyzed 
using descriptive statistics to determine if change occurred in both areas. Due to the start and 
end date of the Street Academy program (December 2005 - May 2006), the only reasonable pre- 
test data were students attendance and discipline data within the same time frame of the previous 
year (December 2004 - May 2005). 

For each indicator (reading, attendance and discipline) a one way analysis of variance 
(ANOVA) was used to determine any statistically significant difference between the treatment 
and internal control group. ANOVA was also used to determine any statistically significant 
difference between elementary and middle school participants within the treatment group. 
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Formative Evaluation Results 



The formative evaluation sought to understand several aspects of the Street Academy 
program implementation process and program components. This includes information regarding 
the student criteria and referral process, curriculum/ activities, and the Street Academy staff. 
Further, embedded in the evaluation design was a formative assessment process where by the 
program participants were assessed mid-program with respect to non-cognitive data to provide 
feedback to program staff and stakeholders. 

Street Academy Student Referral Process 

Students were referred to attend the Street Academy through a process that included the 
Street Academy coordinator and consultant working in conjunction with school principals. First, 
the Street Academy coordinator/consultant notified the Title I school principals of the program 
and asked for their cooperation in referring students to participate in the Street Academy 
program. The principals were given criteria that students must meet in order to be referred. The 
criteria for participating students were as follows: 

• Enrolled in Title I school 

• Receive Free/Reduced lunch 

• African-American male 

• In grades 4, 5, or 6 

• Have academic difficulty in school 

• Have behavioral difficulty in school 

• Have high absenteeism and tardies 

With respect to the student referral process, priority was given to middle school students who 
are failing sixth grade in language arts and/or math that are in danger of not meeting promotion 
requirements. Further, in light of the high need of students who reside in the Wheatley cluster 
area, and because community support is an important focus of the Street Academy program, an 
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effort was made to ensure that elementary and middle school students who live in the Wheatley 
area were given priority. 

Table 3 presents the percent of participating students who met the Street Academy’s 
participant criteria. Behavior difficult is assessed by having more than one suspension or more 
than 3 referrals. Attendance difficulty is assessed by having more than 3 absences or more than 3 
tardies. Academic difficulty was assessed by obtaining a “Novice” on the fall PAS or Kentucky 
Core Content Test (KCCT) in reading, or scoring in the 1 st , 2 nd or 3 rd stanine on the 
Comprehensive Tests of Basic Skills (CTBS). For middle school students, academic difficulty 
also included failing Language Arts or Math. 

As shown on table 4, all participants are African American males, 92% are enrolled in 
Title I schools and receive free/reduced lunch, and 86 % are in grades 4, 5, or 6. Further, as can 
be seen on the map (in appendix) the majority of Street Academy Students live within the 40210 
zip code area. Specifically, 63% of students reside in the 40210 zip code area, and an additional 
16% reside in neighboring zip code areas that border the 40210 area (i.e. zip codes 40212, 

40211, 40216). This area of the city represents the fifth lowest per capita income in Metro 
Louisville at $12,225 according to the 2000 United States Census. According to the Kentucky 
Population Research, University of Louisville report of Neighborhood Profile of Child Well- 
Being, there are 83 percent single -parent households. Over 40 percent of the parents are not high 
school graduates. 
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Table 4 



Street Academy Target Population Analysis (TV = 38) 



Criteria 


Number 


Percent 


Enrolled in Title I School 


36 


94.7% 


Receive Free/Reduced Lunch 


35 


92.1% 


African-American Male 


38 


100.0% 


Grade 4,5, or 6 


33 


86.8% 


Live in 40210 Zip Code 


24 


63.2% 


Academic Difficulty 


30 


78.9% 


Behavioral Difficulty 


15 


39.4% 


Attendance Difficulty 


27 


71.1% 



Street Academy Curriculum/Activities 

Street academy participants that do not walk to the site (Wheately elementary) are picked 
up by a Street Academy staff member. A good deal of effort of the part of the staff is extended 
to ensure good attendance by all members. This includes phone calls and direct door to door 
contact with many of the households. The home-school coordinator works with participants 
parents/guardians to gain commitment to the program and to motivate the students to attend. The 
Street Academy also held a parent meeting during the course of the program. 

While the Street Academy focuses on four areas (i.e. academic, socio-cultural, 
community support and cultural education) the primary focus of the Street Academy program is 
to increase the literacy level and reading skills of participants. This focus is in alignment with 
the Kentucky Core content standards (4.0 for reading and writing) and complements the JCPS 
district wide literacy initiatives. 

The curriculum used to improve student literacy is the corrective reading model, used by 
many JCPS elementary schools. Corrective Reading is a highly structured and scripted direct 
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instruction reading intervention. The Corrective Reading lessons include instruction of word 



attack, story reading and use of workbook. The lessons are enhanced by pacing, signalizing and 
correcting student responses. The Street Academy teachers were trained in the Corrective 
Reading model techniques and the Corrective Reading materials were utilized. The students were 
tested upon entrance by the Corrective Reading assessment to determine reading level. They 
were then grouped accordingly into three groups. 

The program opens with cultural awareness discussion where in participants are exposed 
to positive African American male role models. The cultural discussions include topics such as 
African American literature, African American history as well as knowledge of other cultures. 
The Street Academy schedule is presented on table 5. This schedule indicates that the majority 
of the time is spent on the academic/literacy component. Additionally, there is a short period of 
time when the students eat a snack and socialize with each other. 

Table 5 



Street Academy Saturday Schedule 



Time 


Activity 


1:30-2:00 


Cultural Awareness Discussion 


2:00-3:15 


Corrective Reading Instruction 


3:15-3:30 


Break/Snack 


3:30-4:30 


Corrective Reading Instruction 



Street Academy Teachers/Staff 

The street academy has an all African American staff that includes one program 
coordinator, one home-school coordinator, two JCPS teachers, and two other staff members who 
assist with various program operations such as transportation and secretarial duties. The Street 
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Academy also has an educational consultant who assists with management/guidance of the 
programs curriculum. 

The Street Academy teachers are both African American males, and have earned a 
master’s degree. One teaches in a JCPS elementary school and has been teaching for ten years, 
while the other has been teaching in a JCPS middle school for two years. The program 
coordinator, an African American male instructs participants as a teacher, in part to reduce 
student/teacher ratio. The program coordinator, who also acts as an instructor, holds a doctorate 
in early childhood education and works with the minority teacher recruitment center at the 
University of Louisville. 

The Home School Coordinator serves as key liaisons between the program operations and 
the participating students. This coordinator maintains close interactions with the students, 
guides, and in some cases, advocates for the students. The coordinator also works with 
Neighborhood Places and Family Resource Youth Services Centers to provide needed services. 
Street Academy Attendance 

Over the course of the Street Academy Program, the attendance rate of participants was 
49.8 (i.e., 7.5 sessions). The minimum attendance rate was 6.2 (i.e., 1 session) and the maximum 
attendance rate was 100% (i.e., 16 sessions). Of the total participants (n = 38), four students 
dropped out of the program, and four students began the program within the last two months of 
the program. With respect to participants who exhibited regular attendance to the program (i.e., 
attendance rate between 56%-100%) the mean attendance rate was 69.6%. 

Mid-Program Formative Assessment 

In order to provide mid-program feedback, available data on participating students were 
analyzed and provided to the Street Academy Coordinator, consultant and other stakeholders. 



20 




The purpose of providing mid-program progress data of the Street Academy participants was to 
inform program staff and other stakeholders about the status of participants with respect to 
outcome measures/indicators. This formative assessment also provided a forum for program staff 
and stakeholders to dialogue about important programmatic matters. 

The formative evaluation stressed that the primary goal of the Academy is to improve the 
academic achievement (with focus on reading) of the participants. However, due to testing 
windows / data result turnaround timeframe, the update only included baseline cognitive data and 
its major focus was on non-cognitive data. 

Attendance and Discipline data for this year (January 1 - March 15 th 2006) were compared 
to the baseline data months (December - May 2004/2005). In order to establish if the program 
was on target with respect to indicators being evaluated, a checklist was provided that indicated 
whether or not the current three month data approximates Vi of the base line data (i.e., Vi of six 
month period = 3 months). The progress checklist (in appendix 2), indicated that that the 
participant referrals were in a “danger” zone (i.e., substantially over Vi of baseline), suspensions 
were in a “caution” zone (i.e., slightly over Vi of baseline) and attendance measures were in the 
“ok” zone (i.e., less than Vi baseline). 

In addition to the checklist and a detailed analysis of non-cognitive data, a list of students 
who were receiving the highest number of disciplinary infractions and students who were 
exhibiting the greatest attendance difficulties were provided to program staff. The student list 
was provided to aid program coordinator and program staff members with identification of 
individual participants who are not making progress so that the students can receive additional 
assistance. 
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Summative Evaluation Results 



The summative evaluation sought to assess the impact of the program on participants 
who attended the Street Academy at least 56% of the time. The analyses compared these 
participants with an “internal” control group who participated in the program for a short period 
of time (between 6% - 46%). This will help determine if a greater “dosage” of the program 
results in greater improvements. 

In order to assess the summative measures (i.e., reading, attendance, discipline) between 
the treatment and “internal” control group of the Street Academy, it is important to establish the 
similarities and differences between the two groups. However, it must be noted that both groups 
met the criteria, were referred to the program, and desired to participate in the program as 
evidenced by at least one attendance to the program. 

As presented in table 6, the treatment and control groups are similar in terms of 
demographics (i.e., school level, household structure and lunch status). No statistically 
significant difference was found between treatment and control groups in terms of level, grade, 
household structure and lunch status. 

In addition to demographic similarities, table 6 also presents mean pre program 
differences in terms of cognitive and non-cognitive measures. Similar to demographic 
comparisons, the groups are similar in non-cognitive and cognitive dimensions. No statistically 
significant difference was found between treatment and control group in terms of pre absences, 
tardies, referrals, suspensions, or fall PAS reading score. As previously mentioned, with respect 
to attendance to the Street Academy sessions, the treatment group mean attendance was 69.6, 
while the control group mean attendance was 23.8. 
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Table 6 



Comparison of Treatment and “ Internal ” Control group Profiles (N = 38) 



Variable 


Treatment ( n=21 ) 
N % 


Control ( n=ll ) 
N % 


School Level 


Elementary 


9 


42.9% 


8 


47.1% 


Middle 


12 


57.1% 


9 


52.9% 


Grade 


Third 


1 


4.8% 


0 


0.0% 


Fourth 


4 


19.0% 


5 


29.4% 


Fifth 


4 


19.0% 


3 


17.6% 


Sixth 


11 


52.4% 


6 


35.3% 


Seventh 


0 


0.0% 


3 


17.6% 


Eighth 


1 


4.8% 


0 


0.0% 


Parent Household Structure 


Single 


14 


66.6% 


13 


76.4% 


Dual 


7 


33.4% 


4 


23.6% 



Funch Status 

Free/Reduced 


20 


95.2% 


15 


88.2% 


Paid 


1 


4.8% 


2 


11.8% 





Mean 


Mean 


Non-Cognitive 


Absences 


3.64 


4.79 


Tardies 


5.52 


4.57 


Referrals 


1.24 


.94 


Suspension 


.10 


.25 


Street Academy Attendance 


69.62 


23.81 


Cognitive 


Fall PAS Score 


479.05 


491.76 



Note: All Participants are African-American Males 
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Academic 



In order to assess the academic improvement of Street Academy participants, the 2005- 
2006 PAS reading data were analyzed for the treatment and control group. Additionally, the 
Gates-MacGinitie reading pre (upon entry) and post (upon exit) test is analyzed to assess changes 
in reading grade equivalency. 

Predictive Assessment Series 

The PAS data was analyzed in three ways. The average growth score change from fall - 
winter and from winter to spring was examined for both middle and high school students. The 
fall test was administered at the beginning of the school year and tests student knowledge from 
the prior year and in this evaluation acts as pretest 1. The winter test was administered in 
December (the start of the program), and thus acts as pretest 2. The spring test was administered 
in late February (2 month prior to end of program) yet is the best available post test measure. 

This analysis allows the examination of pre program differences in terms of growth, thereby 
reducing a history/maturation internal validity threat (Shaddish, Cook, & Campbell, 2002). 

A descriptive comparison between the treatment and control group was conducted to 
determine the number of students in novice, apprentice and proficient for the fall, winter and 
spring. Because growth of individual students is perhaps a more critical indicator of success, the 
change in proficiency level was examined to determine the number of students who either 
increased a level, decreased a level, or experienced no change in level. 

The aggregate growth scores and growth score change between treatment and control by 
school level is presented in table 7 and displayed in figures 1 and 2. These finding show that the 
treatment group experienced a minute amount of growth between the fall and winter PAS, yet 
they collectively obtained an average growth of 3.43 points from winter to spring. In particular, 
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the middle school participants obtained a slightly higher average rate of growth (4.17 points) 
than the elementary school participants (2.44 points) during the treatment months (see figure 2). 

However, while the control group decreased in growth scores between fall and winter, 
they increased by 3.82 points between the winter and spring PAS. Interestingly, unlike the 
treatment group, the greatest increase in absolute scores occurred with the elementary students in 
the control group. With respect to change in growth scores from winter to spring, ANOVA 
indicate that there is no statistically significant difference between the treatment and control 
group. This is likely due to small sample sizes and limited statistical power. 

Table 7 



Treatment vs. Control group on Fall, Winter and Spring Average PAS Reading Growth Scores 



Group 


Fall 


Winter 


Fall-Winter 


Spring 


Winter-Spring 


2005 


2005 


Change 


2006 


Change 


Treatment (n = 21) 


Elementary (n=9) 


49.22 


48.67 


-.55 


51.11 


2.44 


Middle (n = 12) 


39.81 


40.75 


.94 


44.92 


4.17 


Total 


44.05 


44.14 


.09 


47.57 


3.43 


Control (n = 16) 


Elementary (n = 8) 


55.12 


50.63 


-4.49 


55.25 


4.62 


Middle (n = 8) 


46.55 


45.00 


-1.55 


48.00 


3.00 


Total 


50.58 


47.81 


-2.77 


51.63 


3.82 
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Figure 1 

Treatment v.v. Control group on Fall, Winter and Spring Average PAS Reading Growth Scores 
for Elementary school level students 




On this figure associated with the elementary school level participants, it can be observed 
that the treatment group gain (1.9) was about twice than the control group gain (.1) from fall to 
spring. 
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Figure 2 

Treatment vs. Control group on Fall, Winter and Spring Average PAS Reading Growth Scores 
for Middle school level students 




On this figure associated with the middle school level participants, it can be observed that 
the treatment group gain (5.1) was approximately five times higher than the control group gain 
(1.5) from fall to spring. 
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Table 8 shows the comparison of the treatment versus the control group on the number of 
students in each PAS proficiency level across assessment. This indicates that between the winter 
and spring, the treatment group experienced a 29% increase in students scoring at the proficient 
level; while the control group experienced a 12% increase in students scoring at the proficient 
level. Both groups experienced approximately 5% increase in students scoring at the novice level 
between winter and spring. 

Table 8 



Street Academy Treatment vs. Control group on Fall, Winter and Spring PAS reading by 
proficiency levels (N =38) 



Group 


Fall 

2005 


Winter 

2005 


Fall-Winter 

Change 


Spring 

2006 


Winter - Spring 
Change 


Treatment (n = 21) 


N % 


Novice 


7 


5 


-2 


6 


+ 1 +4.8% 


Apprentice 


12 


15 


-3 


9 


-6 -28.6% 


Proficient 


1 


1 


0 


6 


+6 +28.6% 


Control (n = 17) 


N % 


Novice 


5 


3 


-2 


4 


+ 1 +5.9% 


Apprentice 


8 


11 


+3 


8 


-3 -17.6% 


Proficient 


4 


2 


-2 


4 


+2 +11.8% 



Note: n varies slightly due to missed test by l-2students each PAS assessment period 



Assessing the number of students in each proficiency level in fall, winter and spring 
provides a general comparative picture of the number of students who attain the novice, 
apprentice, or proficiency level. However, this analysis does not provide an account of individual 
students’ movement between the proficiency levels. 



28 






Table 9 and Figure 3 presents the treatment versus control group with respect to the 
individual movement of students between proficiency levels from the winter to the spring. The 
analysis shows that the treatment group had a greater number of students who increased in 
proficiency level (33%) than did the control group (20%). Further, the control group had a 
greater percent of students who did not move (67%) than did the treatment group (52%). 

Table 9 



Street Academy Treatment vs. Control group on Winter to Spring PAS reading level change by 
number and % of students (N = 36) 





Treatment (n = 21) 


Control (n = 15) 


Winter to Spring Change 


N 


% 


N 


% 


Increase 


7 


33.3% 


3 


20.0% 


Decrease 


3 


14.3% 


2 


13.3% 


No Change 


11 


52.4% 


10 


66.7% 
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Percent of Students 



Figure 3 

Treatment vs. Control group on Winter to Spring PAS level change by % of students (N = 36) 
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Gates-MacGinitie 



Due to unstable attendance, only 22 students took the Gates-MacGinitie pre and post test 
results. Because the control student and treatment student sample sizes largely differ (Cx = 3; Tx 
= 19), the results presented on table 10 only include students in the treatment group. In order to 
assess the impact of the program on decoding and comprehension, dependent sample t - tests 
were conducted to determine if there were any statistically significant changes in raw scores 
from pre to post test. 

As shown on table 10, there was an overall mean decrease of 2.23 in decoding raw scores 
from pre-test (mean = 21.05) to post-test (mean = 18.82). Dependent sample t-test results 
indicate that this reduction was not statistically significant, t (16) = 1.04, p = .31. However, there 
was an overall mean increase in comprehension of 5.83 from pre-test (mean = 17.58) to post-test 
(23.41). The t-test results indicate that this increase was statistically significant, t (16) = -2.84, p 
= .012. Overall, there was a total mean increase of 4.53 from pre-test (mean = 38.65) to post-test 
(43.18). The t-test results indicate that this increase was not statistically significant, t (16) = - 
1.52, p = .147. 

Table 10 

Gates-MacGinitie Raw Score Results 







Pre-Test 


Post-Test 


Score Change 


Test Component 


N 


Mean 


SD 


Mean 


SD 


Mean 


Decoding 


17 


21.05 


10.58 


18.82 


9.37 


-2.23 


Comprehension 


17 


17.58 


7.88 


23.41 


7.54 


5.83* 


Total 


17 


38.65 


17.28 


43.18 


13.44 


4.53 



Note. * p < .05 
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In terms of grade equivalency, descriptive statistics were used to identify changes in the 
participants mean grade equivalency in decoding, and comprehension skills. Table 11 shows that 
there was a mean decrease of .5 in decoding from pre-test (mean = 3.8) to post-test (mean = 3.3). 
However, there was a mean increase of .4 in comprehension from pre-test (mean = 3.1) to post- 
test (mean = 3.5). Finally, there was an overall total mean increase of .2 from pre-test (mean = 
3.4) to post test (mean = 3.6). Thus, the participants acquired the equivalence of two pupil 
months of reading skill improvement. 

Table 11 



Gates-MacGinitie Grade Equivalency Scores (N = 17) 







Pre-Test 


Post-Test 


Change 


Test Component 


N 


Mean Grade 


Mean Grade 


Mean Grade 




Equivalent 


Equivalent 


Equivalent 


Decoding 


17 


3.8 


3.3 


-.5 


Comprehension 


17 


3.1 


3.5 


.4 


Total 


17 


3.4 


3.6 


.2 



Grade Promotion 

One of the primary objectives of the Street Academy program is to aid sixth grade 
participants with successful grade promotion into the 7 th grade. The Street Academy served a 
total of 17 sixth grade students. Demographic data indicate the 94% (n = 16) of the 6 th grade 
participants receive free/reduced lunch, and 71% (n = 12) live in single parent homes. Data 
collected from the district data base reveal that 100% (n = 17) of the 6 th grade participants have 
been successfully promoted into the 7 th grade. This indicates that the Street Academy had a 
positive impact on the successful grade promotion of the 6 th grade participants. 
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School Attendance 



A comparison of the treatment vs. control attendance data is presented on table 12. The 
attendance data indicate that both the treatment and control group experienced an increase in the 
number of absences from last year (December - May 2004/2005) with this year (December - 
May 2005/2006). However, with respect to tardies, while both groups decreased the number of 
tardies, the treatment group had a larger decrease (- 49) than did the control group (-28). The 
ANOVA comparing change in both attendance measures indicate no statistically significant 
difference between the treatment and control group. This is likely due to small sample sizes and 
limited statistical power. 

Table 12 



Treatment vs. Control group on Attendance Measures (N = 36) 



Measure 


Treatment (n = 21) 
N 


Control (n = 14) 
N 


Absences 


Pre 


77 


67 


Post 


97 


95 


Change 


+20 


+28 


Tardies 


Pre 


116 


64 


Post 


67 


36 


Change 


-49 


-28 



Looking at the number of absences and tardies can be misleading due to the presence of 
outliers (i.e., students who incurred the majority of absences or tardies). Thus, this analysis takes 
in to account the recidivism issue. As such, table 13 provides the number and percent of students 
who increase, decrease or experience no change in attendance indicators from pre to post. A 
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slightly larger percent of the treatment group (75%) increased in their absences than did the 



control group (64%). Conversely, a slightly smaller percent of the control group (35%) decreased 
in their absences than did the treatment group (38%). The change by percent of students for 
tardies indicates a similar pattern to that of change in absences. Percent of change in absences for 
both groups are displayed in figure 4 and percent of change in tardies are displayed in figure 5. 



Table 13 



Treatment vs. Control group on Pre to Post change by number and percent of students for 
Attendance Measures (N = 35) 



Measure 


Treatment (n = 21) 
N % 


Control (n = 14) 
N % 


Absences 


Increase 


12 


75.0% 


9 


64.3% 


Decrease 


8 


38.1% 


5 


35.7% 


No Change 


1 


4.8% 


0 


0.0% 


Tardies 


Increase 


8 


38.1% 


4 


28.6% 


Decrease 


12 


57.1% 


7 


50.5% 


No Change 


3 


14.3% 


3 


21.5% 
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Figure 4 

Treatment v.v. Control group on change by percent of students for absences 
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Figure 5 

Treatment v.v. Control group on change by percent of students for tardies 
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School Discipline 

Analysis of discipline data indicates that the treatment group had a greater increased in 
the collective number of discipline referrals (+32) than the control group (-5) when comparing 
baseline (December - May 2004/2005) to treatment year (December - May 2005/2006). As 
shown on table 13, the treatment group also increased in the total number of suspensions (+5) 
than did the control group (0). When considering the data presented on table 14, two 
considerations must be noted. Firstly, there is a good deal of recidivism with respect to both 
referrals and suspensions. Secondly, the sample size is larger in the treatment group. With 
respect to comparing change in treatment vs. control in both discipline measures, ANOVA 
indicate no statistically significant difference. This is likely due to small sample size and limited 
statistical power. 

Table 14 



Treatment vs. Control group on Discipline Measures (N = 35) 



Measure 


Treatment (n = 21) 
N 


Control (n = 14) 
N 


Referrals 




Pre 


26 


15 


Post 


58 


10 


Change 


+32 


-5 


Suspensions 




Pre 


2 


4 


Post 


7 


4 


Change 


+5 


0 



36 





In light of the recidivism and differential sample sizes, the discipline data was examined 
by looking at the number and percent of students who increased, decreased, or experienced no 
change in terms of number of referrals and suspension received. As shown in table 15, a slightly 
larger percent of the treatment group increased the number of referrals (38%) than did the control 
group (28%). However, the percent of treatment participant that decreased in number of referrals 
(14%) was double than that of the control group (7%). In terms of suspensions, a larger percent 
of the treatment group (19%) increased in suspensions than did the control group (7%). Percent 
of change in referrals for both groups are displayed in figure 6 and percent of change in 
suspensions are displayed in figure 7. 

Table 15 



Treatment vs. Control group on Pre to Post change by number and percent of student receiving 
disciplinary infractions (N = 36) 



Measure 


Treatment (n = 21) 
N % 


Control (n = 14) 
N % 


Referrals 


Increase 


8 


38.1% 


4 


28.6% 


Decrease 


3 


14.2% 


1 


7.1% 


No Change 


10 


47.6% 


9 


64.3% 


Suspension 


Increase 


4 


19.0% 


1 


7.1% 


Decrease 


1 


4.7% 


3 


21.4% 


No Change 


16 


76.2% 


10 


71.4% 
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Figure 6 

Treatment vs. Control group on change by percent of student receiving referrals (N = 36) 
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Figure 7 

Treatment v.y. Control group on change by percent of student receiving suspensions (N = 36) 
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Participant Survey 

In order to assess the participants’ perceptions of the Street Academy program, a survey 
was administered at the end of the program. All participants who took the survey were part of the 
“treatment” group. The survey was on a four point scale (i.e., 1 = disagree; 4 = agree). 

As shown in table 16, students responded positively to all survey components. 
Participants responded most positively to statements concerning the Street Academy being 
helpful with school (overall mean = 3.92). These items included statements about the Street 
Academy work helping with students reading, writing, listening, behaving and getting along with 
others in school. Of all the survey components, though relatively high, the lowest mean 
agreement among participants was with respect to the Street Academy teachers (overall mean = 
3.56). Within this component, the lowest rating was with respect to the statement about teachers 
helping when needed (mean = 3.11). However, also within this component, participants 
responded in complete agreement (mean = 4.00) to the statement that their Street Academy 
teachers so a good job. 
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Table 16 



Street Academy Participant Survey Results 



Question 


N 


Mean 


SA Program 


1. helps me learn 


9 


3.89 


2. learning as much as I can at SA 


9 


4.00 


3. happy attending SA 


8 


3.88 


4. work I do in SA is interesting to me 


9 


3.89 


5. work I do in SA will help me do better in school 


9 


3.78 


Total 




3.88 


SA Teachers 


6. my SA teacher does a good job 


9 


4.00 


7. my SA teacher gives me help when I need it 


9 


3.11 


8. My SA teacher shares w/ parents what I’m learning 


9 


3.67 


9. 1 know what my SA teacher expects me to do 


9 


3.56 


10. My SA teacher expects me to learn a great deal 


8 


3.13 


11. My SA teacher lets me know how I am doing each week 


9 


3.89 


Total 




3.56 


SA program helps me in school 


12. The work I do at SA makes me a better reader 


8 


4.00 


13. The work I do at SA makes me a better writer 


8 


4.00 


14. The work I do at SA helps me listen in school 


9 


4.00 


15. The work I do at SA helps me behave well in school 


8 


4.00 


16. The work I do at S A helps me get along with others 


8 


3.63 


Total 




3.92 


Learning in SA 






17. My SA helps me learn about different cultures 


8 


4.00 


18. My SA helps me learn about Black history 


7 


4.00 


19. The SA helps me feel proud of being an AA male 


9 


3.67 


Total 




3.89 



40 






In an effort to evaluate the impact of the program on participants self esteem, the survey 
has additional questions that sought to measure participants feelings about themselves 
retrospectively (i.e. before the Street Academy vs. at the end of the program). All students 
surveyed were part of the “treatment” group. This portion of the survey was on the four point 
scale (i.e., 1 = disagree; 4 = agree) and used both positive and negative statements in order to 
guard against agreement bias. In order to assess the impact of the program dependent sample t- 
test were conducted to determine if there was any statistically significant change in participants’ 
perception of self. 

As shown on table 17, there was an overall mean increase of 1.34 in students’ perception 
of themselves as doing a good job in school from before the program (mean = 2.33) to the end of 
the program (mean = 3.67). Of all the items, this showed the largest mean increase. Dependent 
sample t-test reveals that this increase is statistically significant, t (8) = -2.41, p = .042. The 
second largest mean increase (0.63) was in participants agreement that they liked themselves 
from before the program (mean = 3.25) to the end of the program (mean = 3.88). A dependent 
sample t-test indicated that this increase was not statistically significant, t (7) = -1.66, p = .140. 
Appropriately, the largest mean decrease (-0.34) was in participants agreement that they wished 
they were different from before (mean = 2.67) to the end of the program (mean =2.33). Although 
this was not statistically significant, this change reflects that participants agreed with the 
statement less at the end of the program and indicated an increase in acceptance of self and/or 
increase in self-esteem. 
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Table 17 



Retrospective Survey results 



Question 




Mean 




N 


Before SA 


Now 


Change 


20. 1 don’t do very well in school 


9 


3.11 


3.33 


0.22 


21. 1 am doing a good job in school 


9 


2.33 


3.67 


1 . 34 * 


22. 1 think I am a good student 


9 


2.88 


3.28 


0.40 


23. 1 am not a very good student 


9 


3.00 


3.00 


0.00 


24. 1 wish I were different from the way I am 


9 


2.67 


2.33 


- 0.34 


25. 1 like myself 


8 


3.25 


3.88 


0.63 


26. 1 like myself just the way I am 


8 


3.50 


4.00 


0.50 



* p < .05 



Cost Analysis 

The cost analysis includes examination of the Street Academy Phase I budget and cost of 
program per student. The budget analysis includes program component costs. This includes 
personnel, equipment/facilities, and resources, contractual and indirect costs. The description of 
the budget components including the grand total of program cost is presented in table 18. 

Table 19 shows the cost per student analysis. As shown in table 19, the Street Academy 
served 38 students over the course of 16 Saturday sessions. The cost per student for the duration 
of the program is $945.37. When considering the number of Saturday sessions held, the cost per 
session per student is $59.08. An additional cost analysis that only took into account the students 
who attended the program 56% or more of the time (n = 21) is also shown on table 19. This 
indicates that for the “treatment” students, the cost per student of the program is $1,710.68 and 
the cost per session is $106.92. 
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Table 18 



Street Academy Phase I Budget 



Budget Component 


Amount 


Personnel 


Program Coordinator (1) 


$1,629.18 


Home - School Coordinator (1) 


$1,103.13 


Academic Teachers (2) 


$6,321.42 


Extra Service Pay 


$5,515.65 


Benefits 


$2,715.00 


Total 


$17,284.38 


Resources/Supplies 


Transportation 


$2,500.00 


classroom materials, printing, postage 


$2,740.00 


Food 


$3,150.00 


Cultural arts 


$3,000.00 


Total 


$11,390.00 


Contractual 


Educational Consultant (1) 


$3,150.00 


Indirect Costs 


$4,100.00 


Total 


$7,250.00 


Program Total 


$35,924.38 



Table 19 



Street Academy Phase I Cost Analysis 



Total Program Cost 


$35,924.38 


Number of Students Served 


38 students 


21 “full treatment” 
students 


Cost Per Student 


$945.37 


$1,710.68 


Number of Sessions 


16 


16 


Cost of Session Per Student 


$59.08 


$106.92 
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Discussion 



The findings of this evaluation reflect positively on Phase I of the Street Academy 
program. The formative results suggest that the Street Academy is serving its target population: 
African American males in high poverty neighborhoods, who attend Title I schools and are 
exhibiting academic, behavioral and attendance difficulties. Further, the Street Academy is 
serving students who attend 15 different Jefferson County Public Schools. An all African 
American staff consisting of one program coordinator, two JCPS teachers, one educational 
consultant, and one home-school coordinator have provided a variety of services including 
rigorous literacy instruction, individualized student attention, with an additional focus on 
psychosocial/behavioral development and cultural enrichment. 

As is typical with most evaluations of new programs, the summative evaluation produced 
mixed results both in academic and non-academic outcomes. When compared to the control 
group, a higher percent of the Street Academy participants increased their proficiency level as 
measured using the PAS reading test scores. Although the Street Academy participants increased 
their PAS growth score between winter and spring, the control group had a slightly higher 
increase in scores. Assessment of change in specific reading skills from the start of the program 
to the end via the Gates-MacGinitie test revealed that the participants who experienced a high 
dosage of the program (i.e., < 56%) experienced a statistically significant increase in 
comprehension skills. Further, with respect to grade equivalency, the treatment participants 
acquired the equivalence of two months of reading skills improvement. 

One of the key objectives of the Street Academy was met in that 100% of the 6 th grade 
participants were successfully promoted into the 7 th grade. Regarding non-cognitive indicators, 
Street Academy participants decreased in their number of absences and over half experienced a 
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decline in the number of tardies between the baseline and treatment time periods. In addition, the 
percent of treatment participant doubled the decrease in number of referrals when compared to 
the control group. Nevertheless, with respect to disciplinary infractions, the Street Academy 
participants had a larger total increase in the number of suspensions than the control group. 

Regarding participant perceptions about the program, the treatment participants’ survey 
responses indicated that they perceived the Street Academy was helping them read, write, listen, 
behave and get along with others in school. Moreover, the survey revealed that the program 
increased their self-esteem in general and as well as increased their self-esteem as a student. 
Limitations 

First, it must be noted that this program has only been in operation for six months, and 
met one day per week (Saturday) for 3 hours. Because programs in their infancy are not likely to 
yield striking improvement in outcomes coupled with the short period of time the program has 
run, the program length and dosage may not have been sufficient. Lengthening the program to 
one full school year and including activities during the school week may produce improved 
outcomes. 

Second, with respect to the cognitive data, due to test turnaround, at this time only two 
cognitive measures were used to assess gains in reading (i.e., PAS and Gates-MacGinitie). 
Further, the spring PAS test was administered in late February (2 month prior to end of program) 
yet is the best available post test measure, so has the “post” test in PAS been administered after 
the full dosage of the treatment may yield stronger gains for the treatment group. Due to the 
nature of the Kentucky Core Content Test (KCCT) assessment calendar (with respect to grades 
tested each year), only a limited number of participants have KCCT reading scores. Thus, there 
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was not a sufficient sample size that would allow us to conduct statistical analyses for this 
measure. Future research should incorporate evaluation of participants state assessment scores. 

Finally, one major shortcoming must be noted with respect to the non-cognitive 
measures. Due to the mid year start of the program, the only available pretest measure was the 
same months of the previous year. A good percent of participants were in the 6 th grade this year, 
and thus were in the midst of the transition from elementary to middle school. There is a good 
possibility that changes in discipline and attendance data are reflective of other factors that occur 
during this transition period. Additionally, the sample sizes overall were small and differed from 
the treatment and control groups. 

Recommendations 

First, though the program is shown to be working with its intended target population, the 
process by which students are selected to participate may be improved through the generation of 
a student list by the Research Department. This would provide a precise and accurate student list 
identifying students who are experiencing the most intense academic, behavioral and attendance 
difficulties that also meet the other program criteria. This would further eliminate the 
discretionary component of the referral process that may unintentionally overlook students who 
may be in the most need. 

Second, in light of the importance of providing early intervention coupled with the 
finding of accelerated improvement of elementary students in the program, the Street Academy 
may want to consider targeting more elementary students. 

Third, a greater amount of interaction and communication between program staff and 
school staff/teachers where the participants attend may assist with additional monitoring of 
student progress throughout the school year. This interaction would also aid in student perception 
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of importance of the program and may increase the individual attention they receive inside and 
outside of school. Formalizing this component with a teacher/staff/student contract would be 
useful. 

Fourth, strengthening the relationship between the program staff and the participants’ 
parents would be advantageous for both the program and the participant families. Including a 
strong parental participation component into the program may also improve the attendance rate. 
This could include parent orientation sessions and several parent meetings and focus groups. 

Fifth, with respect to programmatic issues, lengthening the program to one full school 
year may produce improved outcomes as this would allow for a greater “dosage” of the 
treatment. This recommendation also lends support for implementation for Phase II of the Street 
Academy program that extends the time spent with participants from one day a week (Saturday) 
to after school during the weekday. Additionally, the program may want to include a formalized 
socialization component that would focus on the importance of understanding the social skills 
necessary for success in school. This component may also include active dialogue between 
participants and staff regarding “real world” issues facing African American males and strategies 
on dealing with and/or overcoming them. 

Sixth, when a program involves several components, implementation of program 
elements is a challenge. With this reflection in mind, it may be beneficial for the Street Academy 
coordinator to utilize a program implementation checklist. In addition to being a useful tool for 
the program coordinator and evaluators, it would aid in ensuring that the program components 
are executed as planned. 

Finally, due to the drop out of several students who are in high need of intervention, it 
would be beneficial as well as ethical to follow up with students who dropped out or who will 
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not be eligible for the program. This may include following up with parents as well as with the 
school staff at the school they attend. This could also include seeking out services in the 
community they live in that may assist them and their families. 
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Table 1 



Indicator progress checklist 



Indicator 


Vi Baseline 


Current 


Position 


Cognitive 


PAS 




Not Available* 




Gates/M 




Not Available* 




Non-Cognitive 




Referrals 


24 


52 


CAUTION 


4.5 w/o outliers 


20 w/o outliers 


Suspensions 


2 


3 


Absences 


58 


56.5 


OK 


Tardies 


79.5 


44 


OK 


Surveys 


Student 


N/A 


4.0 avg. 


OK 


Staff/Teacher 


N/A 


4.94 


OK 
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Table 2 



Attendance change from Baseline to current year (N = 29) 





Dec-May 

04/05 


Jan-March 

2006 


Change 


% 

Change 


Tardies 


Elementary 


86 


28 


-58 


-67.4% 


Middle 


73 


16 


-57 


-78.1% 


Total 


159 


44 


-115 


-72.3% 


Absences 


Elementary 


43 


15.5 


-27.5 


-64.0% 


Middle 


73 


41 


-32 


-43.8% 


Total 


116 


56.5 


-59.5 


-51.3% 



Table 3 

Discipline data change from baseline to Jan-March 2006 (N = 29) 





Dec-May 

04/05 


Jan-March 

2006 


Change 


% 

Change 


Referrals 










Skipping class 


2 


0 


-2 


-100% 


Bus disturbance 


2 


18 


+16 


+ 12.5% 


Disruptive behavior 


8 


8 


0 


0% 


Fighting/Striking student 


8 


5 


-3 


-37.5 


Verbal conflict/bullying 


2 


3 


+1 


+50% 


Refusal to follow Directions 


17 


0 


-17 


-100% 


Violation of class rules 


5 


0 


-5 


-100% 


Other 


4 


18 


+14 


+350% 


Total Referrals 


48 


52 


+4 


+8.3% 


Total Referrals w/out outliers 


9 


20 


+11 


+122.2 


# students receiving referrals * 


6 


13 


+7 


+116.7% 


Suspensions 










Elementary 


2 


0 


-2 


-100% 


Middle 


2 


3 


+1 


+50% 


Total Suspensions 


4 


3 


-1 


-25% 
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